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(54) Integrated keyboard and pointing device system with automatic mode change. 

(57) A computer system employs a keyboard that 
includes an integrated pointing device coupled 
to a pointing key. The system has a typing mode 
of operation and a pointing mode of operation. 
In the pointing mode, the pointing key is used to 
enter pointing information, thereby obviating a 
discrete pointing device such as a mouse. The 
system may be switched between the typing 
mode and the pointing mode explicitly by pre- 
determined commands. Alternatively, the sys- 
tem may be switched into the ■ pointing mode 
implicitly by monitoring and timing keyboard 
events following a pointing key press. 
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BACKGROUND OF THE INVENTION 

The QWERTY keyboard is entrenched as the pre- 
ferred device for typing alpha-numeric data into a 
computer. Various apparatus and methods are known 
for pointing operations, such as selecting text, on a 
CRT display. Devices for that purpose include the 
mouse, joystick, step keys and text keys. Keyboards 
are known which include a joystick, joydisk or other 
pointing device mounted on the keyboard. However, 
a separate mouse has been shown to be the most 
effective pointing device with respect to operating 
time, error rate and the like. This accounts for its wide- 
spread acceptance in the computer industry. 

Use of a conventional keyboard and a discrete 
mouse for entering typing and pointing information, 
respectively, into a computer require physical and 
mental disruptions that significantly reduce a user's 
productivity. For example, typing on a keyboard and 
pointing with a mouse require the user frequently to 
move the hands back and forth between the keyboard 
and the mouse. One research paper reports that it 
takes about 0.36 seconds to move a hand from the 
keyboard onto a mouse and additional time to adjust 
the grasp for operating the mouse buttons. Time to 
return to the keyboard must be considered as well. 
The data suggests that the total time spent moving to 
and from the mouse is greater than one second per 
occurrence. See S.K. Card, et al, Evaluation of 
Mouse, Rate-Controlled Isometric Joystick, Step 
Keys, and Text keys for Text Selection on a CRT 
(Xerox Palo Alto Research Center), published in 
Ergonomics , Vol.2, No.8, 601-613 (1978). 

These movements between keyboard and mouse 
are disruptive physically because of the distances and 
side to side arm motions typically required. These 
movements are disruptive mentally because of the 
time the movements take, because of the dramatic 
shift in physical activity (typing on a keyboard is very 
unlike pointing with a mouse), and because of many 
mental and physical steps required to perform the 
movements. Each additional step requires physical 
and mental effort and is an opportunity for error. 

U.S. Patent No. 4,712,101 describes a cursor 
positioning device which may be positioned on a 
keyboard below the space bar. Such a device has 
been announced under the trade name Isopoint. The 
Isopoint device includes a finger or thumb actuated 
roller coupled to a rotary-shaft encoder for indicating 
position change in a first direction. The roller rests in 
a sliding cradle which drives a second encoder. Such 
a device is awkward to operate, especially for 
diagonal cursor movement, because of the required 
combination of rolling and sliding actions. 
Additionally, the sliding cradle has fixed end points 
which impose discontinuities in its operation. 

Another pointing device which can be embedded 
into a conventional keyboard for cursor control is the 



OmniPoint tm cursor controller, announced by Osiris 
Technologies, Inc., Nevada City, Ca. Omnipoint 
essentially includes a miniature joystick and 
associated interface electronics. The joystick may be 

5 mounted in a keyboard adjacent the standard array of 
keys. Its use of course requires moving the hand away 
from the usual typing position. 

Both the Isopoint and Omnipoint devices include 
an embedded switch so that the user can press the 

w device downward (into the keyboard) to emulate a 
mouse button "click' , or dragging operation. Such 
devices therefore can emulate at best only a single 
button mouse. 

U.S. Patent No. 4,680,577 to Straayer, et al. 

15 shows a multipurpose keyswitch disposed within a 
keyboard array for controlling cursor movement on a 
CRT display and for character entry. One of the stan- 
dard alpha keys is replaced with the multipurpose 
keyswitch. An additional keyswitch is suggested for 

20 activating the cursor positioning capabilities of the 
multipurpose keyswitch. The '577 patent does not dis- 
close a practical way to implement and use such a 
system. Additionally, that patent does not address 
how to input any pointing event information, for 

25 example mouse button actions ("clicking"), so that 
system cannot substitute for a mouse. What is needed 
is a practical method of allowing a user to type and 
point without moving the hands away from the usual 
typing position and without dramatically changing the 

30 physical activity. 

Prior art methods of pointing and typing assumed 
these disruptions and incorporated them as limi- 
tations in computers, keyboards, pointing devices, 
and software. Typing and pointing were perceived as 

35 distinct and irreconcilable activities. 

As a result, there is considerable duplication of 
hardware and software. For example, the buttons on 
a mouse are required because pointing has been per- 
ceived as very different, mentally and physically, from 

40 typing. Since pointing is so different, the hardware 
and software are duplicated, reinforcing the sepa- 
ration. For example, the buttons on a mouse are dup- 
lications because there are plenty of keys on the 
keyboard that generally are not being used while the 

45 mouse is being used. What is needed is a pointing 
device system that eliminates the restrictions and the 
duplications of present systems, while maintaining 
the same software interface to application programs 
for complete compatibility. 

50 Many known pointing devices also have physical 

limitations on the number of buttons they can have. 
These limitations severely limit the range of actions 
the user can take while pointing. One commercial pro- 
duct that attempts to add additional buttons, called the 

55 PowerMouse 1 00, integrates a two-button mouse with 
forty programmable keys, resulting in a large, cumber- 
some device that is difficult to use. What is needed is 
to allow the user to keep the hands on the keyboard 
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while pointing, and simultaneously allow the user to 
input other pointing related data or pointing events, 
presently input by mouse buttons, but again without 
moving the hands from the usual typing position. 

Another problem overlooked in the prior art is that 
of visually locating the cursor image on a display 
screen. Each time a user stops typing and grasps a 
mouse to point, he has to search the display screen 
visually for the cursor, or actuate the pointing device 
in such a way that the moving cursor will be noticed. 
These methods are time consuming and may be dis- 
ruptive. If the cursor image is mostly offscreen, it can 
be extremely difficult to locate the cursor. In some 
cases, just one pixel is showing. With some software, 
moving the cursor onto certain areas of the screen 
indicates the desire for a particular action to be taken. 
If the user does not wish this action to be taken, they 
must perform additional steps to cancel the implied 
indication. What is needed is for the user to 
immediately detect the location of the cursor on the 
screen at the time pointing begins. 

Known methods of repositioning a cursor in res- 
ponse to signals from a pointing device do not com- 
pensate for the many different modes and resolutions 
of available display systems. Two programs that con- 
figure the display system differently may exhibit diffe- 
rent apparent cursor repositioning responses to the 
same pointing device signals. What is needed is a 
method of cursor repositioning that compensates for 
changes in the display system so that the same action 
performed by the user with the pointing device will 
produce a similar result independent of changes in the 
display system. 

Adjusting cursor speed is another clumsy and 
time consuming action. Known methods of changing 
cursor speed introduce discontinuities that seriously 
disrupt the user's work flow. As a result, many people 
simply tolerate an inappropriate and therefore ineffi- 
cient cursor speed because it is too cumbersome and 
too burdensome to change it. 

For example, a typical mouse requires that the 
user first stop the application program in progress, 
and then execute a special mouse speed changing 
program, where the speed is specified by typing in a 
number. The resulting cursor speed cannot be obser- 
ved until the application program is restarted. The 
user may find another adjustment is necessary and 
have to repeat the process. 

Other known methods involve pressing keyboard 
keys and mouse buttons simultaneously, but these 
schemes are not continuous and interfere with the 
operation of certain application programs. What is 
needed is to afford a user fast, interactive ways to con- 
trol cursor speed without significant interruption of 
work in progress. 



SUMMARY OF THE INVENTION 

It is an object of the present invention to allow a 
computer user to type and to point without removing 
5 the hands from the home row (asdf-jkl;) of the 
keyboard, and without dramatically changing the 
physical activity. 

Another object of the invention is to reduce the 
restrictions in operation and the duplications of 
w hardware and software found in present computer 
systems, while maintaining a standard software inter- 
face to existing programs for compatibility. 

Another object of the invention is to allow the user 
to keep the hands on the keyboard while pointing, 
15 simultaneously allowing the userto conveniently input 
pointing related data or "pointing events", again with- 
out moving the hands from the usual typing position. 

A further object of the invention is to enable a 
computer user to immediately detect the location of 
20 the cursor on the display screen whenever a pointing 
operation begins. 

Yet another object is to compensate for variations 
in display systems when repositioning a display cur- 
sor so that a given action performed by the user with 
25 the pointing device will produce a similar result inde- 
pendent of variations in display systems. 

According to the present invention, a single, inte- 
grated keyboard system is provided for inputting all 
typing and pointing information to a computer without 
30 moving the hands from the usual typing position. One 
of the keyboard keys, called the pointing key, has sen- 
sors coupled to it to acquire pointing direction data 
from lateral displacement of the pointing key. 

The new system has a typing mode of operation 
35 and a pointing mode. In the typing mode, key codes 
corresponding to actuated keyswitches are forwarded 
to the operating system in the usual fashion. In the 
pointing mode, operation of the entire keyboard is 
changed. All of the keys become available for new 
40 functions. 

One or more of the keys is assigned as a pointing 
event key, for inputting information which would come 
from mouse buttons in a segregated system. Other 
keys may be assigned various meanings for modify- 
45 ing operation of the system, such a cursor speed con- 
trol, macros, etc. 

Cursor movement is implemented in the pointing 
mode by reading sensor data from the pointing key, 
mapping the data to form cursor displacement data, 
so and scaling the displacement data according to a 
speed index. Keys may be assigned for changing the 
speed index, and therefore the apparent cursor 
speed, at any time. This allows speed control interac- 
tively and without leaving an application program. 
55 The cursor mapping may be done by algebraic 

manipulation of the sensor data according to a pre- 
determined tracking algorithm, or by lookup tables. 
The new system also allows detecting the display 
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mode, for example character or graphics mode, and 
altering cursor speed accordingly to optimize control. 

According to another aspect of the invention, a 
user's intent to change from typing to pointing oper- 
ations (and vice versa) can be inferred from the user's 
actions at the integrated keyboard. In response to a 
pointing key press during typing mode, the system 
enters an intermediate "wait and see" mode, and def- 
ers processing the pointing key press to determined 
whether the user intends to begin pointing. 

Subsequent keyboard actions are timed and tes- 
ted to make that determination. A conclusion is 
reached, and the system either switches to pointing or 
resumes the normal typing mode, within one-half sec- 
ond. This process is unobtrusive, if not transparent, to 
the user. In use, to switch to pointing mode, the user 
needs merely begin pointing. To switch back, the user 
simply stops pointing and starts typing. 

The foregoing and other objects, features and 
advantages of the invention will become more readily 
apparent from the following detailed description of a 
preferred embodiment which proceeds with reference 
to the drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is an illustration of a prior art computer sys- 
tem that includes a keyboard and a mouse pointing 
device. 

FIG. 2 is a high level block diagram of a prior art 
computer keyboard. 

FIG. 3 is a prior art computer keyboard proces- 
sing flow chart. 

FIG. 4 is a prior art high level system block diag- 
ram of the software elements involved in processing 
keyboard data. 

FIG. 5 is a control flow diagram for a prior art com- 
puter keyboard interrupt handler system. 

FIG. 6 is a high level system block diagram of 
prior art software elements involved in processing 
pointing device data. 

FIG. 7 is a system block diagram of a prior art 
computer system of the type illustrated in FIG. 1. 

FIG. 8 is user flow chart illustrating the control 
flow of a computer user's decisions and actions while 
using a prior art computer system of the type illus- 
trated in FIG. 1. 

FIG. 9 is a high level block diagram of the 
hardware elements of an integrated keyboard and 
pointing device system according to the present 
invention. 

FIG. 1 0 is a flow chart showing the high level pro- 
cessing control flow for the integrated computer 
keyboard and pointing device system of FIG. 9. 

FIG. 1 1 is a system block diagram of a computer 
system that includes an integrated keyboard and 
pointing device of the type illustrated in FIG. 9. 

FIG. 12 is a flow chart of integrated interface 



software for implementing the integrated keyboard 
and pointing device of FIGS. 9-1 1 . 

FIG. 13 is a user flow chart illustrating control flow 
of a computer user's decisions and actions while 
5 using a computer system of the type shown in FIGS. 
9-12. 

FIG. 1 4 is a user flow chart illustrating control flow 
of a computer user's decisions and actions while 
using an integrated pointing device speed control 
w according to the present invention. 

FIG. 15 is an enlarged, perspective view of a 
space bar on a conventional keyboard modified to 
include a thumb switch in the front face of the space 
bar, 

15 FIG. 16 is an enlarged, perspective view of a pair 

of thumbswitches positioned in front of the space bar 
on a keyboard. 

FIG. 17 shows a time line that includes a series 
of six discreet states used for timing keyboard events. 
20 FIG. 1 8 is a flowchart for a timer interrupt used for 

transitions to a following state after conclusion of pre- 
determined time intervals. 

FIG. 19 shows a modified portion of a keyboard 
interrupt flowchart. 
25 FIG. 20 is a flowchart of a method for handling a 

keyboard interrupt during timing state 0. 

FIG. 21 is a flowchart of a method for handling a 
keyboard interrupt during timing state 1. 

FIG. 22 is a flowchart of a method for handling a 
30 keyboard interrupt during timing state 2. 

FIG. 23 is a flowchart of a method for handling a 
keyboard interrupt during timing state 3. 

FIG. 24 is a flowchart of a method for handling a 
keyboard interrupt during timing state 4. 
35 FIG. 25 is a flowchart of a method for handling a 

keyboard interrupt during the pointing mode of oper- 
ation. 

DETAILED DESCRIPTION OF THE PREFERRED 
40 EMBODIMENT 

The present invention is best understood with 
reference first to the prior art. FIG. 1 illustrates a prior 
art computer system. The prior art system includes a 

45 computer 28, a keyboard 24 for inputting character 
data to the computer, a pointing device 26 for inputting 
graphic data to the computer; and a display terminal 
29 for display of data output by the computer. The 
"arrow" 25 points to the row of keys containing the 

so home row keys of the keyboard 24. The keyboard is 
coupled to the computer by a communications link 36. 
The pointing device is coupled to the computer by a 
pointing device communications link 23. The pointing 
device 26 may be, for example, a mouse that includes 

55 one or more mouse buttons 27. The pointing device 
generally is used to position and to move the cursor, 
as more fully explained below. 

FIG. 2 is a block diagram of the keyboard 24 
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employed in the system of FIG. 1. The keyboard 24 
includes an array 32 of keyswitches, of which keys- 
witch 30 is an example. The keyswitches in array 32 
are arranged in rows and columns to facilitate scan- 
ning the keyboard. A microprocessor system 34 
includes hardware for electrically interfacing the keys- 
witch array 32 to a microprocessor within the system 
34. 

In operation, microprocessor system 34 scans 
the rows and columns of array 32 to detect closures 
and openings of keyswitches. Upon detection of a 
closure or an opening of a keyswitch, the microp- 
rocessor system 34 transmits a code unique to the 
closed or opened keyswitch via a keyboard communi- 
cations link 36 to the computer (28 in FIG. 1). 

Referring now to FIG. 3, a high level processing 
control flow diagram for the keyboard of FIG. 2 is 
depicted to more specifically illustrate the creation 
and transmission of keyboard data to the computer. In 
FIG. 3, an initialize step 42 tests microprocessor sys- 
tem (34 in FIG. 2) and initializes the keyboard 
software state. 

After initialization, a keyboard scan loop 56 
begins by scanning the keyboard 44. Scan Keyboard 
44 sequentially scans the rows and columns of the 
keyswitch array to detect changes in the state of the 
keyswitches. After Scan Keyboard 44 is completed, 
the results are checked in block 46. Control passes to 
an Auto Repeat block 52 if there have been no 
changes in the keyswitches. If there has been a 
change, a control passes to Determine Key block 54 
to identify the affected keyswitch. 

Determine Key block 54 analyzes the changes to 
determine what binary key codes, if any, should be 
transmitted to the computer. Send Key Code, block 
60, transfers the key codes. In the typical case of a 
single keyswitch press and release, one unique key 
code corresponding to the keyswitch will be sent 
when the key is pressed and another code when the 
keyswitch is released. 

The software represented by Auto Repeat 52 
determines if it is time to automatically repeat sending 
the key code if the keyswitch remains pressed After 
the key codes, if any, have been sent, control returns 
via loop 56 to begin the next keyboard scan. 

Referring to FIG. 4, a block diagram of typical 
prior art keyboard software interface is depicted. In 
this system, a computer keyboard 74 generates 
binary key codes when the operator presses keys- 
witches as described above. Binary key codes are 
transmitted to the computer via a keyboard communi- 
cations link 36. A keyboard interrupt handler 72 reads 
the key codes, processes them, and places corre- 
sponding character codes into a keyboard queue 70. 
Generally, the keyboard interrupt handler 72 is sim- 
ple, fast interrupt level software. 

An application program 62 is coupled to operating 
system keyboard services 66 via an interface 64. In 



response to requests by the application program, the 
operating system removes character codes from the 
keyboard queue 70, via another interface, operating 
system keyboard queue interface 68, and sends the 
5 character codes to the application program 62. 
Operating system keyboard services 66 may examine 
the character codes to check for operating system 
requests such as application program termination. 
Interfaces 64 and 68 are usually bi-directional to 

10 enable the application program 62 to "put back" 
character codes as needed. 

Operation of the keyboard interrupt handler 72 is 
illustrated in greater detail in FIG. 5. Referring to FIG. 
5, a Read Hardware block 76 reads the key codes 

75 from the keyboard interface hardware. Each such key 
code is checked to see if it represents any of the Shift, 
Caps Lock, Control and ALT keyswitches. These spe- 
cial cases are tested in blocks 78, 82, 86 and 90, re- 
spectively. For each test, if the test is negative, control 

20 is passed to the next test in the sequence shown. If 
all of the special case tests are negative, control pas- 
ses to Map to Char block 94 for mapping the keycode. 
If any of the tests are positive, control is passed to 
update a corresponding one of status blocks 80, 84, 

25 88 and 92, indicating whether the special key was 
pressed or released. 

Map to Char block 94 uses a look-up table or 
tables to map the key code to a character code. The 
mapping includes the current states of the shift, caps 

30 lock, control, and ALT keys. This operation is a simple 
pass-through mapping. The resulting character code, 
if any, is passed on to Enqueue block 96. Generally, 
if the key code represents a keyswitch release, it is not 
passed on to Enqueue software 96. Enqueue 96 

35 inserts the resulting character code into a keyboard 
queue (70 in FIG. 4). After a status update, or 
Enqueue operation, as the case may be. control 
returns to the interrupted program in the usual way, 
Return block 98. 

40 FIG. 6 is a block diagram illustrating the operation 

of typical prior art pointing device software. Here, an 
application program 62 communicates with operating 
system (OS) pointing device services software 104. 
The application program passes pointing device con- 

45 trol information, such as initial position, cursor shape, 
cursor visibility, etc., and receives pointing device 
state information such as position and button presses. 

The OS services 104 in turn communicates with 
pointing device driver interface software 108 in 

so accordance with interface specifications 106. Driver 
interface software 108 converts and formats the par- 
ticular pointing device's data to conform to the inter- 
face 106. Pointing device state data, including 
information such as cursor shape, visibility, location 

55 and pointing device button status, are maintained in 
a set of memory locations called state data 114. 

The pointing device driver interface software 108 
communicates with state data 114 via an interface 
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112 and with low level pointing device software 118 
via an interface 110. The low level software 118 also 
communicates with state data 114 through an inter- 
face 116. 

In operation, the low level pointing device 5 
software 118 reads data from pointing device 
hardware 120 via the pointing device communications 
link 23. Software 1 18 modifies state data 1 14 and noti- 
fies the driver interface 108 as needed. The low level 
software 118 may also include software to move the w 
cursor on the display terminal (29 in FIG. 1) 

FIG. 7 is a software block diagram of a complete 
computer, keyboard and pointing device system such 
as that illustrated in FIG. 1 . It essentially consists of a 
combination of keyboard software (see FIG. 4) and 15 
pointing device software (see FIG. 6). Accordingly, 
the details of operation need not be repeated here. 

It is noteworthy that, from the lowest to highest 
levels, the data and control paths between the appli- 
cation program 62 and the keyboard 74 are separate 20 
and distinct from the data and control paths between 
the application program and the pointing device 120. 
This separation of the keyboard and pointing device 
interfaces reflects the segregation of keyboard and 
pointing device operations in the user's mind. Useful 25 
improvements in the state of this art require careful 
consideration of a computer user's thoughts and 
actions, described next. 

FIG. 8 is a "user flow chart" for using a typical 
computer system of the type illustrated in FIGS. 1-7. 30 
The figure diagrams the control flow of a computer 
user's decisions and actions while using such a sys- 
tem. Referring to FIG. 8, a primary loop 140 begins 
with a decision whether to point or type, represented 
by block 1 22. If the decision (mentally) is to type, a test 35 
134 is performed to see if the user is typing already. 
Test 134 must be performed to make sure the hands 
and the rest of the user's body are positioned for typ- 
ing. If not T the user moves their hands to a typing posi- 
tion, (block 136) generally on the home row of the 40 
keyboard, to begin typing. Once the hands and body 
are properly positioned, the user presses keyboard 
keyswitches to enter alpha-numeric character 138, 
usually repeatedly. 

If the decision is made to point, the user tests 45 
(mentally) to see if he or she is pointing already, 124. 
This must be done to make sure the hands and the 
rest of the user's body are positioned for operating the 
pointing device. If pointing already, the user has 
merely to continue. If not, the user must move their 50 
hands to the pointing device 126, and then find the 
cursor on the display screen, 128. 

The step of moving a hand to the pointing device 
includes whatever is required to begin operating the 
pointing device if the user's hands have been typing 55 
on the keyboard. Generally, this includes moving a 
hand off of the home row of the keyboard to where the 
pointing device is located, grasping the pointing 



device, positioning the fingers to operate the pointing 
device event buttons, and repositioning the arm into 
a comfortable position. 

Finding the cursor on the display screen 128 may 
be as simple as remembering where the cursor was 
last seen, orcoincidentally having it in the field of view 
Quite often, it involves operating the pointing device 
and looking for a moving object on the display screen, 
thereby wasting time and effort. 

Once a hand is on the pointing device and the cur- 
sor has been located, the pointing device may be 
operated to perform two types of operations - "point- 
ing" and "pointing events". Pointing refers to moving 
the mouse 130 to reposition a cursor. Pointing events 
are indicated by pressing pointing device buttons 132, 
which may include pressing, holding, or releasing 
selected pointing device buttons. Pointing events are 
used to select text, pick a menu item, and many other 
functions. Typical pointing device operations may 
involve repeated pointing and pointing events. When 
there is a break in activities, or when a change in 
activities is required, the user effectively returns to the 
point or type decision 122 via the primary loop 140. 

Overview of Integrated Keyboard and Pointing 
Device System 

FIG. 9 is a simplified block diagram of an inte- 
grated keyboard and pointing device (integrated 
keyboard) according to the present invention. The 
new integrated keyboard includes an array 232 of 
keyswitches, of which keyswitch 230 is an example. 
Direction sensors 238 are embedded in the array 232 
to detect pointing information. Preferably, the direc- 
tion sensors are force sensors coupled to one of the 
keyswitches in the array to form a multi-purpose keys- 
witch called the pointing key. For example, where the 
direction sensors are coupled to the J key, the user 
can input pointing information by pressing on the J key 
in the desired direction. An example of such a multi- 
purpose keyswitch is disclosed in U.S. 4,680,577, 
assigned to Home Row, Inc., incorporated herein by 
this reference. The J key is preferred as the pointing 
key because it is located beneath the user's right 
index finger when the user's hands are positioned on 
the home row of the keyboard. How the integrated 
system distinguishes between a user inputting direc- 
tion information versus merely entering the letter J is 
described below. 

A smaller and less expensive embodiment would 
include four force-sensitive resistors (FSRs) coupled 
to the J key, one FSR for sensing force in each of the 
four directions in a plane. Analog to Digital conversion 
circuitry 240 includes electronic circuitry for trans- 
forming the FSR resistances, analog information, to 
representative digital signals. Analog to digital con- 
version is known and therefore will not be described 
in detail. 
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Analog to Digital circuitry 240 is coupled to the 
microprocessor system 234 via a path 242. Microp- 
rocessor system 234 includes hardware for electri- 
cally interfacing the array 232 and A/D circuitry 240 to 
a microprocessor (not shown) within the system 234. 

The integrated keyboard and pointing device has 
at least two distinct modes of operation; a typing mode 
for entering alpha-numeric character data, and a 
pointing mode for entering pointing information, such 
as would be input in the prior art by a separate pointing 
device such as a mouse. Pointing information 
includes pointing direction information and pointing 
event information. 

In general, the microprocessor system 234 scans 
the keyswitch array 232 to detect closures and open- 
ings of key switches. Upon detection of a closure or an 
opening of a keyswitch, system 234 transmits a code 
unique to the actuated keyswitch over a keyboard 
communications link 236. System 234 also deter- 
mines whether the sensors 238 must be scanned by 
examining the sequence of keyswitch closures and 
openings, or by receiving a message from the com- 
puter system. If so, a sensor scan is performed to 
acquire pointing direction information from the sen- 
sors 238, and the pointing direction information also 
is transmitted over link 236. A separate physical con- 
nection to the computer for sending pointing direction 
information is not required. 

FIG. 10 is a flow chart of operation of the inte- 
grated keyboard and pointing device of FIG. 9. Refer- 
ring to FIG. 10, an initialization step 42 tests the 
microprocessor system 234 (FIG. 9) and initializes the 
keyboard software state. Next, keyboard scan loop 56 
begins with scanning the keyboard to determine the 
current states of the keyswitches. 

After scanning 44 is completed, the results are 
checked in step 46 to determine if any keyswitch state 
changes have occurred. If there have been no 
changes, control passes to an auto repeat step 52. 
Auto repeat step 52 functions as described above with 
regard to FIG. 3. If there have been changes, control 
passes to determine which keyswitch state changed, 
step 54. Step 46 does not indicate a changed condi- 
tion until multiple keyboard scanning passes have 
been made for software keyswitch debounce. 

Determine key step 54 analyzes the changes to 
determine what binary key codes, if any, should be 
transmitted to the computer, step 60. In the typical 
case of a keyswitch press and release, one unique 
key code corresponding to the keyswitch will be sent 
when the keyswitch is pressed and another when the 
keyswitch is released. 

After the keyswitch changes, if any, have been 
processed, a decision 144 determines if the keyboard 
operating mode must be changed between typing and 
pointing modes. This check for mode change may 
consist of examining keyswitch presses and releases, 
checking for commands from the computer via the 



communications link or checking for activation by the 
user of other sensors on the keyboard, such as 
thumbswitches (described below). 

FIG. 16 illustrates a pair of thumbswitches 218 

5 positioned below the space bar 214 of a keyboard 212 
for indicating an operating mode change. The 
thumbswitches 218 are operated by sliding them in 
the plane of the keyboard generally towards the space 
bar. The thumbswitches are conveniently placed and 

w operated at such an angle as to allow the natural 
movement of the thumbs towards the other fingers to 
provide the actuating effect. 

The mode of the integrated keyboard and point- 
ing device may be changed from typing to pointing by 

15 operating a thumbswitch. The effects on the pointing 
mode are determined by the next keyboard key 
pressed. For example, pointing mode may be entered 
by operating a thumbswitch, and when the J key is 
pressed to operate the pointing device, the 

20 thumbswitch may be released. When the J key is 
released, the keyboard returns to typing mode. Thus, 
the thumbswitch only indicates a possible entry into 
pointing mode, and not necessarily the exit from point- 
ing mode. The thumbswitches 218 allow fast, natural 

25 operation without moving the hands away from the 
home row of the keyboard. 

An appropriate mode change is made if indicated. 
Next, a test for pointing mode 146 is performed to 
determine if pointing device processing must be done. 

30 If the current mode is typing, control returns via loop 
56 to again scan the keyboard 44. 

If the current mode is pointing, the system next 
reads the direction sensors to acquire pointing direc- 
tion data, and processes that data, step 148. Sensor 

35 data processing may include amplification, filtering, 
and nonlinear transformations. The sensor data pro- 
cessing that occurs at this point is simple and fixed in 
nature. Additional processing in the computer system 
remains to be done. When the sensor data processing 

40 is completed, the processed sensor data is sent to the 
computer 1 50. In an alternative embodiment, raw sen- 
sor data could be sent to the computer for processing, 
either in real-time or in packets. After the processed 
sensor data is sent, control passes over loop 56 to 

45 again scan the keyboard, and the foregoing process 
is repeated. 

FIG. 11 is a block diagram of a computer system 
that includes the integrated keyboard of FIGS. 9-10. 
At the application program (62) and operating system 

so (66, 104) levels, this figure is similar to FIG. 7. At the 
lowest interface level however, all user input is via 
common "integrated interface software" 162. This 
novel structure enables additional control to be perfor- 
med and new features implemented, as further des- 

55 cribed below. By intercepting keyboard and pointing 
device data at the lowest levels, new operations may 
be provided which are entirely transparent to the 
application program. Thus, the application program 
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"thinks" the system includes a mouse. 

The computer system of FIG. 1 1 operates as fol- 
lows. The integrated keyboard 166 sends binary 
encoded key code and pointing device sensor data 
over a communications link 236 to integrated inter- 
face software 162. The integrated interface software 
162 a special interrupt handler for separating the data 
stream from the integrated keyboard into keyboard 
and pointing device data streams. Integrated interface 
software 162 includes key code-to-character code 
mapping software, and additional pointing device data 
processing software, including software to move the 
cursor on the display screen. 

A key code processing portion of the integrated 
interface software 162 operates in a manner similar to 
keyboard interrupt handler 72, with additional 
software to detect and honor keyboard/pointing 
device mode change requests. Software 162 inserts 
character codes into the keyboard queue 70. Operat- 
ing system keyboard services 66 reads the queue 70 
via operating system keyboard queue operations 68 
upon request by the application program 62. The 
applications program is linked to the OS keyboard ser- 
vices via an interface 64. 

A set of memory locations called integrated state 
data 158 includes everything contained in the pointing 
device state data 114 (FIG. 6), as well as additional 
state data to keep track of operating modes (for 
example typing and pointing) and new features (for 
example variable cursor tracking controls, described 
below). Interface 156 and particularly interface 160 
likewise have additional controls and features as 
compared to the corresponding interfaces 112 and 
116 found in a typical pointing device (FIG. 6), as 
required in view of the new integrated interface 
software described below. 

Driver interface block 152 provides a standard 
pointing device interface as required by OS (Operat- 
ing System) pointing services 104 and the application 
program 62. Both the interface between the appli- 
cation program and the operating system, and the 
interface 106 between the operating system and the 
driver interface 152 are similar to a typical pointing 
device system so that operation of the integrated sys- 
tem is transparent to the OS and to the application 
program. 

Integrated interface software 162 communicates 
with driver interface 152 as shown by path 154. 

Integrated Keyboard/Pointing Device Interface 
Software 

FIG. 1 2 is a control flow diagram for the integrated 
interface software 162. It includes all the processing 
performed by a typical keyboard interrupt handler 
(see FIG. 5), indicated by dashed line 99. Additionally, 
integrated interface software 162 includes software to 
detect and honor keyboard/pointing device mode 



change requests, convert pointing device sensor data 
into changes in cursor position, emulate pointing 
device buttons using keyboard keyswitches, modify 
pointing device operations using keyboard keys- 

5 witches as controls, and provide integrated typing and 
pointing macro capabilities from a pointing device 
using keyboard keyswitches (188). Each of these 
features is described in turn below, in the order of con- 
trol flow shown in FIG. 12. 

w A Read Hardware block 76 reads binary key 

code/pointing device data provided over the com- 
munications link 236 (FIG. 11). Once the data has 
been read, Check Mode block 167 determines the 
present mode (for example, typing or pointing) of the 

is integrated keyboard/pointing device. Control trans- 
fers to Change Mode? 1 69 if the present mode is typ- 
ing, and to Change Mode? 170 if the present mode is 
pointing. The typing/pointing mode encompasses all 
other keyboard states (shift, caps lock, control, alt), so 

20 that typing/pointing mode becomes a new mode on 
top of all these conventional keyboard modes, not 
merely in addition to them. 

Typing Mode 

25 

Change Mode? 169 determines if the acquired 
key code data indicates that the user wishes to 
change the mode of the integrated keyboard from typ- 
ing to pointing. This determination may include check- 

30 ing for a unique predetermined code, or a more 
complex test based on multiple previous codes and 
other keyboard modes (shift, caps lock, control, alt) as 
well. If the test determines that integrated keyboard is 
to be changed from typing to pointing mode, the 

35 requested change is performed in Change Mode 168. 
Change Mode 168 includes setting flags in state data 
1 58 (FIG. 1 1 ), changing the visual appearance of the 
cursor and acknowledging the mode change to the 
user. After the mode change is completed, nothing 

40 more needs to be done until further data is received, 
so control is passed to return from interrupt 98. 

If Change Mode? 169 determines that no change 
of mode is necessary, the key code is treated as it 
would be treated by a typical keyboard interrupt han- 

45 dler. Thus, tests for Shift Key 78, Caps Lock key 82, 
Control Key 86, and Alt Key 90 are performed. The 
status of each of these special keys 80, 84, 88 and 92, 
respectively, are updated as required. Map Key Code 
to Character Code 94, Enqueue Character Code 96, 

so and Return From Interrupt 98 all operate as in the typi- 
cal keyboard interrupt handler. 

Pointing Mode 

55 If Check Mode 167 determines that the integrated 

keyboard device is in pointing mode, a test 1 70 is per- 
formed to determine if the user wishes to begin typing 
again. This test may include examining a single data 
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item, or a more complicated test based on conven- 
tional keyboard modes (shift, caps lock, control, alt) 
and previous binary key code/pointing device data 
items. If Change Mode? 170 determines the mode is 
to be changed to typing mode, the requested change 
is performed by Change Mode block 172. 

Change Mode 172 may include setting flags in 
state data 1 58, changing the visual appearance of the 
cursor and acknowledging the mode change to the 
user. After the requested mode change is effected, 
nothing more needs to be done until further data is 
received, so control is passed to Return 98. 

If Change Mode? 170 determines that the mode 
is to remain pointing, the next test performed on the 
data is to test for cursor positioning data, labeled 
Pointing Data? block 174. This test determines 
whether the data is pointing device positioning data or 
key code data. This test may include checking flags 
and counters in state data 1 58 as well as testing bits 
within the data itself. 

If Pointing Data? 174 determines that the data is 
cursor positioning data, Update (cursor) Position 
block 1 76 is performed to move the cursor on the com- 
puter display screen, update the cursor position in the 
state data 158, and notify the driver interface 152 
(FIG. 11) of the changes. Update Position 176 may 
include cursor tracking algorithms to convert the cur- 
sor positioning data into changes in cursor position. 
Upon completion, control is transferred to return from 
interrupt 98. 

Cursor movement is effected as follows. The 
pointing data are read (76) and the cursor reposi- 
tioned periodically, for example, as driven by a 
keyboard clock interrupt operating at approximately 
18-20 Hz. The "cursor speed" apparent to the user 
actually is proportional to the magnitude of changes 
in cursor position each time the cursor is repositioned, 
and the frequency of repositioning. A change in the 
clock speed will of course result in a proportional 
change in apparent cursor speed, so the keyboard 
clock rate of a system should be taken into account in 
cursor tracking. 

Accordingly, to effect high apparent cursor 
speed, a given input force is mapped to a relatively 
large change in cursor position. To "slow" the cursor, 
changes in position are scaled down. 

The update position or tracking software 176, in 
addition to responding to pointing data from the 
keyboard, can also consider implicit speed controls 
implied by the state of the pointing event keys as ref- 
lected in the state data. 

The tracking software also takes into account the 
explicit speed modification settings, controlled by 
modification software 184, the display resolution, as 
determined by the standard means for the particular 
computer and display system, and the display mode 
(character cells or graphics pixels) as determined by 
the standard means for the particular computer and 



display system. 

One operative example of implementing the cur- 
sor tracking software uses multiple predetermined 
look-up tables. For instance, a set of graphics mode 

5 look-up tables take into account everything except the 
implicit pointing event key speed controls and the exp- 
licit speed modification settings controlled by modify 
operation 184. Two sets of two graphics mode tables 
would be provided: a horizontal and a vertical table 

w set for use when the pointing event keys are not 
pressed, and another horizontal and vertical set for 
when the pointing event keys are pressed. 

The partially processed data from the communi- 
cations link 236 is used as an index into the approp- 

15 riate table. The result of this table look-up operation 
is a cursor displacement, which may be multiplied by 
a scaling factor determined by the explicit cursor 
speed controls in modify operation 1 84. After this mul- 
tiplication, the displacement is used as a change in 

20 the horizontal or vertical component of the cursor 
position. The new cursor position is then calculated by 
adding the displacement to the present position. After 
both the horizontal and vertical coordinates have 
been recomputed, the cursor image is erased from the 

25 present position and redrawn at the new position. 
Analogous table sets may be provided for character 
cell display modes. 

An advantage of using different tables depending 
on the state of the pointing event keys is that, with 

30 most application software, the pointing event keys in 
the released position indicates nothing important is 
happening. In this situation, the user generally desires 
to move the cursor as quickly as possible, to reduce 
the cursor repositioning time. 

35 When a pointing event key is pressed, the user is 

indicating something specific and generally desires to 
move the cursor in a more controlled manner. Speed 
is important when the event buttons are not being 
pressed, and control is important when an event but- 

40 ton is being pressed. In practice, it has been found 
that an approximately quadratic force-to-apparent 
cursor speed mapping table works best for fast cursor 
positioning, and that an approximately linear mapping 
table works best for more controlled cursor position- 

45 ing. 

Pointing Events 

"Pointing events" are input by a user of pointing- 
50 related information, usually other than cursor posi- 
tioning or directional information. For example, in the 
prior art. pointing events include pressing and releas- 
ing mouse buttons, also referred to as "clicking" the 
mouse buttons. Pointing events may serve to select 
55 an item associated with the current position of the cur- 
sor. Or, a pointing event may signal a request to move 
previously selected text to the current location of the 
cursor. 
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According to thepresent invention, one or more 
of the keyswitches in the integrated keyboard are 
designated for emulating pointing events. These are 
called pointing event keys. When the integrated 
keyboard is in the pointing mode, actuation of one of 5 
the pointing event keys is interpreted as a pointing 
event. This allows a user to input pointing events with- 
out removing their hands from the keyboard. Prefer- 
ably, the D and F keys, or other keys on the home row 
of the keyboard, are so designated for emulating 10 
pointing events, so that all typing and pointing oper- 
ations can be conducted from the normal typing posi- 
tion. This feature yields substantial time savings and 
ergonomic advantages to a user of the integrated 
keyboard. Pointing event emulation is implemented in 75 
the preferred embodiment as follows. 

Referring again to FIG. 12, if test 174 determines 
that the binary key code/pointing device data is not 
pointing device data, the data is then known to be key 
code data. In that case, control passes to test 178, to 20 
test the data for pointing device button emulation. 

If the data represents any of the keyboard keys- 
witches designated to emulate pointing device but- 
tons, the key code data is passed on to block 180 
where the key code data is converted to changes in 25 
pointing device button status and stored in state data 
1 58 (FIG. 11). This information may be used in various 
ways. For example, it may be helpful in some appli- 
cations to temporarily freeze cursor motion for a pre- 
determined period of time in response to a pointing 30 
event, to minimize the effects of inadvertent cursor 
movement during the pointing event. 

Software 180 thus emulates the use of pointing 
device event buttons such as mouse buttons. 
Because nearly any or all of the keyswitches in the 35 
array may be designated to emulate pointing device 
buttons, the number of keyswitches so designated is 
limited only by the number of keyswitches on the 
keyboard. This feature provides for more flexibility in 
pointing operations as compared to a two or three but- 40 
ton stand alone pointing device. When the emulation 
actions 180 have been completed, control is passed 
to block 184, where implicit changes to the operation 
of the pointing device are performed, such as chang- 
ing the cursor tracking algorithm depending on the 45 
state of the emulated pointing device buttons. 

Alternative Interface Software for Implicit Mode 
Change 

50 

The processes corresponding to FIG. 12, blocks 
167,168,169,170 and 172 illustrate a system in which 
mode change is effected explicitly by predetermined 
keyboard events defined for that purpose. 

The integrated keyboard and pointing device sys- 55 
tern can be arranged to provide for implicit mode 
change, i.e., to switch from the typing mode to the 
pointing mode of operation in response to user 



actions at the keyboard other than actions explicitly 
directed to a mode change as described above. A 
user's intent to change mode can be inferred from the 
users actions at the keyboard. For example, a user 
intending to begin pointing might depress the pointing 
key and hold it down for a period of time longer than 
is usually encountered in typing. During typing mode, 
these actions can be used to infer that a change to the 
pointing mode is desired. 

Whether or not a mode change is appropriate, 
however, will depend upon the user's actions after the 
pointing key press, and the timing of those actions. 
Therefore, responsive to a pointing key press in typing 
mode, the system goes into an intermediate or wait 
and see mode to subsequently determine whether to 
finally switch to pointing mode or resume the normal 
typing mode. This decision-making process, illus- 
trated in FIGS. 17-25, is implemented so as to be sub- 
stantially transparent to the user. 

We have found that the distributions of time inter- 
vals between keyboard events (usually key presses) 
for typing versus pointing are distinctly different. 
Graphically, these distributions would resemble a 
bimodal probability curve. When typing, the amount of 
time a key is held down is typically less than 100 mil- 
liseconds. When pointing, however, the amount of 
time the key is held down is typically not less than 500 
milliseconds (mostly due to the fact that the user's 
tracking response time is around 400 milliseconds). 

Also, when typing, it is common for a second key 
to be pressed or released soon after any given key is 
pressed, for example within about 200 milliseconds. 
When pointing, however, at least about 300 mil- 
liseconds elapses before a pointing event button is 
pressed. At least that much time is necessary for rep- 
ositioning the cursor, even over a short distance. 

To provide for implicit mode change, the system 
processes a pointing key press detected in typing 
mode as follows. First, the pointing key press is inter- 
cepted in that, initially, it is not processed as a usual 
typing event. (Typically, that would be mapping the 
key press and enqueing it). Rather, the system begins 
monitoring a time interval elapsed since the pointing 
key press, and monitors the keyboard to detect a sec- 
ond keyboard event. After a sufficiently long time pas- 
ses without detecting a second keyboard event, intent 
to point is inferred, and the system switches to the 
pointing mode of operation. This is a safe conclusion 
after about 400 milliseconds. 

Conversely, if a second keyboard event is detec- 
ted very soon after the pointing key press, within 
approximately 100 milliseconds, it is concluded that 
the user intends to remain in the normal typing mode. 
Accordingly, the system processes first the pointing 
key press and then the second keyboard event as nor- 
mal typing events. The wait and see delay of a few 
hundred milliseconds is not disruptive to the user. 
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Implementation of Implicit Mode Change 

Implementation of the methods just described 
and of methods of determining whether to change 
mode when a second keyboard event arrives within 
the 1 00-400 millisecond wait and see window are des- 
cribed next. 

The "J" key will be referred to herein interchange- 
ably with the "pointing key" as any key on the 
keyboard may be coupled to force sensors to serve as 
a pointing key. 

When the "J" key is pressed, the chording 
keys' (SHIFT, CONTROL and ALT) status are 
checked. If any of these are being held down, it is 
assumed that the user wishes to type, and the "J" key 
pressed is passed on normally. If none of the chording 
keys are pressed, it cannot yet be determined whether 
the "J" was pressed with the intention of typing or 
pointing. 

The fact that the "J" key was pressed is stored 
internally, and a timer is started. The timer may be 
implemented in a variety of ways, for example, using 
a hardware timer in the keyboard controller, or a 
software loop, or some interrupt signal from a host 
processor. Preferably, in response to the "J" key 
press, the keyboard is instructed to start sending 
pointing sensor data packets. Receipt of the sensor 
data packets is used as a timer interrupt to clock the 
timer. The packets are read in and parsed, but are not 
used to cause the cursor to move, at least initially. 

FIG. 17 shows a timeline for implementing an 
implicit mode change feature. As shown, there are six 
states numbered 0 through 5. Each state extends for 
an interval of approximately 100 milliseconds after the 
conclusion of a next preceding state. These times are 
approximate and may be varied to meet individual 
needs. State 0 is the normal typing mode. Responsive 
to a pointing key press, the State advances to State 
1 and the timer interrupt is enabled to begin monitor- 
ing the time elapsed since the pointing key press. 

FIGS. 18-25 are flowcharts of the implicit mode 
change methodology. In all of these flowcharts, the 
following conventions apply. Control starts at the top 
of the flowchart and flows generally toward the bot- 
tom. Decisions are indicated by a question mark (?) 
in a process or step description. Control flows to the 
right from an affirmative decision. 

FIG. 18 is a flowchart for the timer interrupt. This 
process is not executed unless the timer is enabled 
(started). First, a counter is incremented. Next, the 
current state is checked. For each state that has a fol- 
lowing state, the value of the count is checked to see 
if the current time period (State) has elapsed. If the 
current period is concluded, the state transitions to the 
next state. Additionally, special actions are performed 
in some cases further described below. 

FIG. 19 shows a new or modified portion of the 
keyboard interrupt routine. It is assumed here that any 



handshaking protocol and any sensor data packets 
have already been handled. It is also assumed that 
during the intermediate or "wait and see" states, point- 
ing key presses detected after the first pointing key 

5 press (results of auto-repeat) are ignored. As shown 
in the flowchart, in response to a keyboard interrupt, 
control passes to a respective one of FIGS. 20-25 that 
corresponds to the present timing state. For example, 
in response to a first keyboard interrupt during State 

w 0, control passes to a process represented by the 
flowchart of FIG. 20. 

Timing State 0 

15 Referring now to FIG. 20, the system checks to 

determine if the keyboard event is a pointing key 
press. If the first keyboard event is a pointing key 
press, the system tests the states of the chording 
keys. If any of the chording keys is also pressed, the 

20 pointing key press is processed normally and the sys- 
tem continues in the typing mode. By the phrase, "pro- 
cessed normally", it is meant that the pointing key 
press is further processed in the same fashion as any 
keyboard event in the normal typing mode. For 

25 example, a keycode corresponding to the pointing key 
press may be enqued for transmission to an appli- 
cation program. The particulars of such a transmis- 
sion are known. 

The user's intent to type is inferred from use of the 

30 chording keys in combination with the pointing key. 
This detects many known typing events and allows 
the auto repeat function to work on the pointing key if 
the user so desires. For example, to repeat lower case 
Js, the user can invoke CAPS LOCK, press and hold 

35 SHIFT, and press and hold "J". Auto repeat will func- 
tion and the system remains in typing mode. 

Referring now to the right side of FIG. 20, if the 
first keyboard event was a pointing key press and 
none of the chording keys was also pressed, the state 

40 is set to state 1, a "wait and see" state. The system 
begins to monitor the time elapsed since the pointing 
key press by enabling the timer. The system also 
starts to acquire sensor scan data from the pointing 
key. 

45 State 1 is a "wait and see" state that corresponds 

to FIG, 21. If a second keyboard event occurs in this 
state, it is inferred that the user intends to continue 
typing. Accordingly, referring to FIG. 21, the system 
enqueues the pointing key press, resets to state 0, 

so stops the timer, stops sensor data acquisition and pro- 
cesses the new (second) keyboard event normally so 
that the system resumes the normal typing mode of 
operation. 

55 Timing State 2 

Three events are continuously monitored, 
namely, the "J" key going back up (pointing key 
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release), other keyboard events, and the timer. Ref- 
erring to FIG. 18, if the system is in state 1, and the 
elapsed time exceeds approximately 100 mil- 
liseconds, the state advances to State 2; the present 
cursor position is saved (stored); and then the cursor 
is allowed to begin moving. 

During state 2, if the timer exceeds a second pre- 
determined time interval, approximately 200 mil- 
liseconds, the state advances to state 3 (see FIG. 1 8). 
On the other hand, if a second keyboard event is 
detected during state 2. the control passes to code 
represented by the flowchart of FIG. 22. 

As shown in FIG. 22, in this case, the system 
resumes the normal typing mode. Specifically, resum- 
ing the typing mode may include creating an indi- 
cation of the pointing key press for further processing 
as a typing event; resetting to state 0; stopping the 
timer; stopping the sensor data acquisition; restoring 
the cursor to the initial position; and, processing the 
second keyboard event normally. 

Timing State 3 

As indicated, if no keyboard eveht occurs during 
timing state 2. the system advances to state 3, and 
continues to monitor the keyboard to detect the next 
keyboard event. Accordingly, if the elapsed time since 
the pointing key press is greater than the second time 
interval, and a second keyboard event has not been 
detected, the system is in state 3. 

In this state, in response to detecting a second 
keyboard event, control passes to code represented 
by the flowchart of FIG. 23. Here, the system tests to 
determine whether the second keyboard event is a 
pointing key release. If the second keyboard event is 
not a pointing key release, the system determines 
whether the second keyboard event is a pointing 
event. If the second keyboard event is a pointing 
event, the pointing event is enqued and the system 
switches to the pointing mode of operation. 

To place this in context, state 3 corresponds to 
the interval from approximately 200 to 300 mil- 
liseconds after the pointing key press. During that 
interval, the likelihood is that the user intends to begin 
pointing, though this is not yet certain. But if the next 
keyboard event is a pointing event, such as pressing 
one of the keys designated as a pointing event key, it 
is then concluded that pointing mode is desired. The 
assumption here is that the likelihood of a user want- 
ing to type a \T followed by an "F". "D". or "S" is lower 
than the user wanting a pointing event. This is con- 
ditioned such that a typing event is assumed below 
200 milliseconds and a pointing event after that. We 
have found that if a key is held down longer than 
approximately 300 milliseconds, it is very unlikely that 
another key would be pressed unless the user inten- 
ded to point. 

Referring again to FIG. 23, if the second keyboard 



event is not a pointing event, or if the second 
keyboard event is a pointing key release, then the sys- 
tem resumes the typing mode of operation. The speci- 
fic steps taken to resume the typing mode of operation 
5 are the same as those mentioned above with regard 
to FIG. 22. 

Timing State 4 

w If the elapsed time is greater than the third time 

interval (300 msec) and a second keyboard event has 
not been detected, the state is advanced to state 4, 
as shown in FIG. 18. If a second keyboard event is 
detected during state 4, control is directed, according 

15 to the keyboard interrupt flowchart of FIG. 1 9, to code 
that corresponds to the flowchart of FIG. 24. 

Referring to FIG. 24, the system tests to deter- 
mine whether the second keyboard event is a pointing 
key release. If the second keyboard event is not a 

20 pointing key release, indicated on the left side of FIG. 
24. the second keyboard event is processed as a 
pointing macro and the state is advanced to state 5, 
i.e., the system switches to the pointing mode of oper- 
ation. In other words, referring to the timeline of FIG. 

25 1 , if the pointing key has been held down for more than 
300 milliseconds (state 4), and a keyboard event 
other than a pointing key release is detected, it is 
assumed to be a pointing macro and the system is 
switched to a pointing mode of operation. On the other 

30 hand, if the event is a pointing key release, corre- 
sponding to the right side of FIG. 24, the system 
resumes the typing mode of operation, in the same 
manner described above. 

3$ Timing State 5 

Referring back again to FIG. 18, if the elapsed 
time since the pointing key press is greater than a 
fourth predetermined interval, about 400 mil- 

40 liseconds, pointing mode is assumed. Thus, the sys- 
tem switches to the pointing mode of operation 
without explicit instruction from the user to do so. This 
corresponds to timing state 5. In timing state 5, a 
keyboard interrupt is handled according to the flow- 

45 chart of FIG. 25. Referring to FIG. 25, in response to 
detecting a keyboard event in this state, the system 
tests to determine whether the event is a pointing key 
release. If not, the new keyboard event is processed 
as a pointing macro. Alternatively, if the keyboard 

50 event is a pointing key release, as shown on the right 
side of FIG. 25, the system switches back to the typing 
mode of operation. Specifically, state is reset to state 
0, the timer is stopped, and sensor scan data acquisi- 
tion is stopped. The system thus resumes the typing 

55 mode of operation as described above. 

The foregoing mode change principles are applic- 
able to systems other than conventional computers 
with integrated keyboards. For example, any elec- 
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tronic apparatus may be equipped with a multi-func- 
tion keyswitch. The apparatus may have a first operat- 
ing mode, analogous to the typing mode of a computer 
keyboard system, in which the multi-purpose keys- 
witch has a binary function (press and release). The 
apparatus may be configured, using the above des- 
cribed methods, to be switched into a second mode 
for acquiring pointing or pointing direction information 
from the multi-purpose keyswitch. This change may 
be effected implicitly as described. 

Interactive User Modification of the Integrated 
Keyboard System 

Referring back to FIG. 12, explicit modification of 
the integrated system, i.e. changes in operation 
requested by a user, are described next. If test 178 
determines that the key code information does not 
represent any of the pointing event keyswitches, con- 
trol is passed to a test 182 to determine whether the 
key code represents a keyswitch assigned to modify 
operation of the pointing device or other aspects of the 
integrated keyboard system. Such keyswitches are 
called modify operation keyswitches. and can perform 
various tasks further described below. If the key code 
represent a modify operation keyswitch, control is 
passed to modification software 184. Modification 
software 184 uses the key code information to modify 
operation of the system as requested. 

In a system of the type described, such modifi- 
cations may include a wide variety of changes to the 
behavior of the system. The effects of some of these 
changes are known in the prior art, but known ways 
to accomplish them are quite different and far less effi- 
cient. Other modifications are applicable only to the 
new. integrated keyboard system. 

To illustrate, it is known to change the apparent 
cursor speed in a mouse system by first exiting the 
application program, running a special program to 
alter cursor speed, and then re-entering the desired 
application program. In the integrated system, a 
single stroke of the keyswitch assigned the cursor 
speed-up (or down) function accomplishes the same 
result. Obviously, the change is made more quickly 
and without significant interruption of the user's work. 
Importantly, the speed control is interactive in that the 
user immediately sees the result of the speed adjust- 
ment. Further adjustment may be made, immediately, 
if deemed necessary. 

Another illustrative modification is changing the 
condition to exit from pointing mode. In other words, 
entering a pointing-lock mode, in which the user has 
to do more than release the pointing key to resume 
typing mode. For instance, pressing the space bar 
while in pointing mode may be arranged to enter a 
pointing-lock mode, so the system does not exit point- 
ing mode, even if the J key is released. The system 
may return to typing mode only when the space bar is 



pressed. 

Yet another use of modify operation keys is to 
lock/unlock or change axes of the pointing device. In 
the past, these changes have been made by special 
5 additional mouse buttons, or by running special 
software at the application level. Making these selec- 
tions with modification keys is much faster and less 
disruptive. 

Another example of a system modifications which 

w may be implemented with modify operation keys is 
changing the cursor tracking algorithm. For example, 
various predefined tracking algorithms may be pro- 
vided, each of which is optimized for a particular appli- 
cation, e.g. word processing, CAD, etc. A single 

15 keystroke selects the best tracking algorithm for cur- 
sor control. After the indicated modify operation is 
completed, control is passed to Return 98. 

Yet another modification is to toggle the inte- 
grated keyboard between right and left handed oper- 

20 ation. For example, in right-handed mode on a 
QWERTY keyboard, the J key is the pointing device 
and the D and F keys are conveniently used as point- 
ing event keys. Upon switching to left-handed mode, 
the F key is the pointing device and J and K keys are 

25 pointing event keys. This feature requires two mul- 
tipurpose keyswitches (disposed at F and J), but 
otherwise is easily implemented in the software by 
simply changing the interpretation of the affected keys 
for pointing mode. Thus, the test for modify operation 

30 keys would look for J and K to detect pointing events 
while in left-handed mode, but look for D and F for the 
same purpose while in right-handed mode. These 
examples merely begin to illustrate the functions 
available with an integrated keyboard system. Virtu- 

35 ally all of them may be implemented at the lower 
software levels, namely the interrupt and driver levels, 
so they are transparent to the application software 
and the operating system. 

Another illustrative use of modify operation keys 

40 is for macros. Macros, used as abbreviations for seg- 
ments of text and/or strings of program commands, 
are known. In the integrated system, macros can 
include a mix of cursor motions, pointing events and 
keyboard strokes. For example, a macro can be set 

45 up to move the cursor, pull down a menu and pick an 
item. This feature can easily be programmed in the 
integrated system environment described herein. 

If the test for pointing device modification (182) 
determines the key code does not represent any other 

so modification request, control is passed to a test for 
macro expansion 188. Macro Expand 188 determines 
if the key code is one that is currently a macro com- 
mand in need of expansion. If so, the macro is expan- 
ded and control is passed to Change Mode? 170 

55 where the expanded data is operated on. 

If Macro Expand 188 determines the key code is 
not one that is currently designated for macro expan- 
sion, control is passed to a test for typing emulation, 
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"Alpha Emulate" block 186. Alpha Emulate 186 deter- 
mines if the key code is currently being mapped back 
to a typing operation, and if so, control is passed to 
test for shift key "Shift"? 78 and key code processing 
continues as if the integrated device mode was typing 5 
instead of pointing. This feature thereby effectively 
changes the device temporarily to typing mode auto- 
matically in response to the user entering typing data. 
The user can resume pointing activity without explici- 
tly changing mode. If test 186 determines the key w 
code is not currently being mapped back to a typing 
operation, control is passed to Return 98 and the key 
code is effectively ignored. 

Details of implementing the foregoing software 
will be apparent to those skilled in the art in view of this 15 
disclosure. The various features indicated as perfor- 
med in the computer, as distinguished from in the inte- 
grated keyboard, preferably are implemented in 
software so that they can be employed on various 
computer systems without modification of the com- 20 
puter hardware. 

Operation of the Integrated System 

FIG. 1 3 depicts a user flowchart for an integrated 25 
keyboard according to the present invention. The fig- 
ure illustrates the control flow of the computer user's 
decisions and actions while using the integrated sys- 
tem. Once the user has begun a task requiring both 
typing and pointing activities, the user must 30 
repeatedly make a mental decision to point or type 
1 22. If the decision is to type, a test to see if the user 
is typing already 134 must be performed to make sure 
the mode of the integrated device is in typing mode. 
If not, a change to typing mode 192 must be perfor- 35 
med before typing can begin. This mode change is a 
simple action that can be performed with the hands in 
the typing position on the keyboard. Once the mode 
is typing, the user proceeds to type 138. 

If the decision 1 22 is made to point, a test to see 40 
if pointing already 124 must be performed to make 
sure the integrated device is already in pointing mode 
If so, user has merely to continue. If not, the user must 
change the keyboard to pointing mode 190 and notice 
the cursor 194. 45 

The switch to pointing mode 190 can be perfor- 
med with the hands in the typing position on the 
keyboard. The cursor is easy to find because, in res- 
ponse to the mode change, the system causes the 
cursor image on the display screen to flash, for 50 
example by intermittently enlarging the cursor image. 
Noticing the change in cursor appearance is particu- 
larly easy because the peripheral vision of the human 
eye is extremely welt adapted at noticing changes in 
brightness and movement. Thus, there is no need to 55 
remember where the cursor was last seen, or coinci- 
dentally have it in the field of view, or to actuate the 
pointing device to detect cursor motion. 



Once the integrated device is set to pointing 
mode, the pointing device may be operated 130 and 
useful actions performed. Pointing events are indi- 
cated by pressing, holding, or releasing keyboard 
keys 196. This aspect of the integrated system allows 
the user to indicate pointing events while the hands 
remain in the typing position. Typical pointing device 
operations may involve repeatedly moving the cursor 
and pressing keyswitches. 

When there is a break in activities, or when a 
change in activities is required, the user effectively 
returns to the decision to point or type 122 via loop 
140. 

Cursor Speed Control 

FIG. 14 is a user flow chart for changing the cur- 
sor speed using the integrated keyboard. It is 
assumed, of course, that the system is in the pointing 
mode. The user starts out by deciding if the cursor 
speed is acceptable 198 or if it should be altered. If the 
present speed is too slow, the user can increase it. 
pressing a predetermined keyboard keyswitch (for 
example, the R key) 204. If the cursor is deemed too 
fast, decelerating the cursor is likewise as simple as 
pressing a predetermined keyboard keyswitch (for 
example, the V key) 206. After the appropriate keys- 
witch is pressed, the user moves the cursor 208 to 
determine the effect of the change, and again decides 
if the speed is acceptable 198. If the speed is still not 
correct, the foregoing steps are repeated. 

Once the cursor speed is correct, control trans- 
fers to Done 210 where any speed modification keys- 
witches that were being pressed are released. This 
process may be continuous: the user has only to 
press a rate control keyswitch, move the cursor until 
the speed reaches the desired rate, and release the 
rate control keyswitch. The user thus obtains instant 
feedback of cursor speed changes without leaving the 
application program and, indeed, without even mov- 
ing a hand away from the typing position. 

Mode Change Switch 

FIG. 15 illustrates modification of the space bar 
on a conventional keyboard in accordance with the 
present invention to distinguish being pressed 
towards the other keys in the keyboard as well as 
down into the keyboard. Space bar 214 is shown in 
the conventional position below and in front of 
example keyboard keys 212. Mode change bar 216 is 
placed in the front face of space bar 214 for ready 
operation by the user's thumbs. When the hands are 
in the typing position, the natural grasping action of 
the thumbs easily brings them to push mode change 
bar 216 further into the front of space bar 21 4, activat- 
ing a mode change keyswitch mounted beneath the 
space bar. 
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Having illustrated and described the principles of 
my invention in a preferred embodiment thereof, it 
should be readily apparent to those skilled in the art 
that the invention can be modified in arrangement and 
detail without departing from such principles. In par- 
ticular, but without limitation, allocation of functions 
between hardware and software is subject to wide 
variation depending upon numerous design consider- 
ations for any particular application. The principles 
disclosed herein can be implemented in many diffe- 
rent combinations of hardware and software, as a 
matter of design choices, without departing from the 
principles of the invention. We claim all modifications 
coming within the spirit and scope of the accompany- 
ing claims. 

Claims 

1 . In a computer system having a type mode of oper- 
ation and a pointing mode of operation, and 
including a keyboard having a plurality of keys 
that includes typing keys, one of the typing keys 
being also a pointing key, and a display screen 
having a cursor, an implicit method of switching 
between typing mode and pointing mode, com- 
prising the steps of: 

during the typing mode of operation, moni- 
toring the keyboard to detect a first keyboard 
event; 

responsive to detecting a first keyboard 
event, testing for a pointing key press; 

if the first keyboard event is not a pointing 
key press, processing the first keyboard event 
normally; and 

if the first keyboard event is a pointing key 
press, monitoring a time elapsed since the point- 
ing key press for determining whether to switch to 
the pointing mode. 

2. A method according to claim 1 further comprising: 

if the first keyboard event is a pointing key 
press, testing the states of the chord ing keys; and 

if any of the chording keys is also pressed, 
processing the pointing key press normally and 
resuming the typing mode of operation. 

3. A method according to claim 1 or 2 further com- 
prising: 

if the first keyboard event was : pointing 
key press, monitoring the keyboard to detect a 
second keyboard event; and 

responsive to detecting a second 
keyboard event, if the elapsed time is less than a 
first predetermined time interval, processing the 
pointing key press normally and then processing 
the second keyboard event normally whereby the 
system resumes the typing mode of operation. 



4. A method according to claim 1 .2 or 3 further com- 
prising: 

if the first keyboard event was a pointing 
key press, monitoring the keyboard to detect a 
5 second keyboard event; and 

if the elapsed time is greater than a first 
predetermined time interval and no second 
keyboard event has been detected, storing an ini- 
tial cursor position, acquiring pointing sensor 
w scan data and moving the cursor responsive to 

the sensor scan data. 

5. A method according to claim 1, 2, 3 or 4 further 
comprising: 

is if the elapsed time is greater than a first 

determined time interval and a second keyboard 
event has not been detected, monitoring the 
keyboard to detect a second keyboard event; 
acquiring pointing sensor scan data; 

20 moving the cursor responsive to the sen- 

sor scan data; and 

responsive to detecting a second 
keyboard event, if the elapsed time is less than a 
second predetermined time interval, resuming 

25 the typing mode of operation. 

6. A method of according to claim 5 wherein resum- 
ing the typing mode includes: 

processing the pointing key press nor- 

30 mally; 

stopping the cursor; and 
processing the second keyboard event 
normally. 

35 7. A method according to claim 1 further comprising: 
if the elapsed time is greater than a first 
predetermined time interval and a second 
keyboard event has not been detected, monitor- 
ing the keyboard to detect a second keyboard 
40 event; and 

responsive to detecting a second 
keyboard event, if the elapsed time is less than a 
second predetermined time interval, determining 
whether the second keyboard event is a pointing 
45 key release; 

if the second keyboard event is not a point- 
ing key release, determining whether the second 
keyboard event is a pointing event; 

if the second keyboard event is a pointing 
50 event, processing the pointing event normally 

and switching the system to the pointing mode of 
operation. 

8. A method according to claim 7 further comprising: 
55 if the second keyboard event is not a pointing 

event, resuming the typing mode of operation; 
and 

if the second keyboard event is a pointing 
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key release, resuming the typing mode of oper- 
ation. 

9. A method according to claim 8 wherein resuming 
the typing mode includes: 

processing the pointing key press nor- 
mally; 

stopping the cursor; and 
processing the second keyboard event 
normally. 

10. A method according to claim 7,8 or 9 further com- 
prising: 

if the elapsed time is greater than the sec- 
ond time interval and a second keyboard event 
has not been detected, monitoring the keyboard 
to detect a second keyboard event; 

responsive to detecting a second 
keyboard event, if the elapsed time is less than a 
third predetermined time interval, determining 
whether the second keyboard event is a pointing 
key release; 

if the second keyboard event is not a point- 
ing key release, processing the second keyboard 
event as a pointing macro and switching the sys- 
tem to the pointing mode of operation. 

11. A method according to claim 10 further compris- 
ing: 

if the second keyboard event is a pointing 
key release, resuming the typing mode of oper- 
ation. 

12. A method according to any one of the preceding 
claims further comprising: 

monitoring the keyboard to detect a sec- 
ond keyboard event; 

responsive to detecting a second 
keyboard event, if the elapsed time is greater than 
a predetermined time interval, switching the sys- 
tem to the pointing mode of operation, whereby 
the system changes mode of operation without 
explicit instruction from a user to do so. 

13. A method according to claim 12 further compris- 
ing: 

responsive to detecting a second 
keyboard event, determining whether the second 
keyboard event is a pointing key release; 

if the second keyboard event is not a point 
key release, processing the second keyboard 
event as a pointing macro; and 

if the second keyboard event is a pointing 
key release, switching the system to the typing 
mode of operation. 

14. In a computer system having a typing mode of 
operation and a pointing mode of operation, and 



including a keyboard having a plurality of typing 
keys; one of which is also a pointing key, a 
method of switching from typing mode to pointing 
mode, comprising the steps of: 

5 during the typing mode of operation, moni- 

toring the keyboard to detect a pointing key press; 

responsive to a pointing key press, enabl- 
ing a timer to indicate a time elapsed since the 
pointing key press; 

w monitoring the keyboard to detect a sec- 

ond keyboard event; and 

if no second keyboard event is detected 
before the elapsed time exceeds a first predeter- 
mined time interval, switching the system to the 

15 pointing mode of operation, whereby the said 

mode change is effected implicitly by a user's 
actions. 

1 5. A method according to claim 1 4 wherein enabling 
20 the timer includes providing a counter and incre- 

menting a counter responsive to receipt of a sen- 
sor scan data packet and switching to the pointing 
mode includes acquiring sensor scan data pack- 
ets from the pointing key. 
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